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ABSTRACT 


The purpose of this study was to adapt and illustrate the use of a 
computer program to score binary patterns of response on a short- 
form predictor test (Electronics Technician Selection Test and the 
General Classification Test) so as to maximize the correlation 
between this predictor and a criterion (the final school grade in the 


Basic Electronics and Electricity School). 
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I, INTRODUCTION 


The Navy has been very much interested in recent years in the 
possibility of using short-form tests to reduce testing time while 
maintaining or even increasing test reliability and validity. 

The advantages of a short-form test are manifold. With a 
short but reliable and valid test the Navy could save thousands of 
dollars in training costs by weeding out, before training even began, 
those individuals who would probably not succeed. ‘The administration 
of the test could be done at a training command, e.g., Naval Training 
Center, San Diego, Bainbridge, etc., or even by a recruiter. For 
example, if an individual desires to be a radioman and talks toa 
recruiter about joining the Navy only if accepted for radioman training, 
it would be advantageous for both the service and that individual if a 
brief test of possibly five to seven minutes’ duration could be admin- 
istered, graded and evaluated on the spot against the individual's 
desires for such a Navy career. With this brief test both the Navy 
and the potential recruit would know, ina relatively short period 


of time, whether the man would succeed in radioman training. 





Il. BACKGROUND 


Moonan (Ref. 1) Rioneered this type of work for the Navy by 
constructing a computer program having the capability of identifying 
combinations of test items that have maximal validity. This program, 
entitled SEQUIN (an acronym for Sequential Item Nominator) first 
selects an item that has highest validity with the criterion. ‘The pro- 
gram then continues to select another item which, when.combined 
with the first, mrodices a two-item test with a higher validity than 
any Other two-item test that includes the first item. ‘This process 
continues until the required number of items is selected and the 
maximum validity for this number of items is obtained. ‘The advan- 
tage of such a program is that a fairly long test, such as the General 
Classification Test (GCT), might be shortened without sacrificing 
validity while test time might be significantly reduced. 

SEQUIN nae shown, repeatedly, that a short-form test is at 
least as ecu of final school grade as its long-form counterpart 
(Ret. 2). Swanson and Rimland (Ref. 3) have found that a short form 
of the GCT, e.g. one-half to one-third of the original length, is even 
more predictive of recruit final achievement (RFAT) than the complete 


f@teha. 
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I. THE PROBLEM 


This study attempts to increase further the predictive validity 
of an already brief test. The method, developed by Dr. R. A. 
Weitzman of the Naval Postgraduate School, Monterey, California, 
is to weight item responses so as to maximize the correlation with the 
criterion. 

On an n-item test where each question is graded to be either 
correct or incorrect, there are 2" different possible patterns of 
correct and incorrect responses. ‘Thus, for example, on a five-item 
test there are 32 possible pattern scores as opposed to six possible 
scores if just the number of correct responses were tallied. 

For example, suppose a three question testis given to a group 
of recruits in an attempt to predict their success in a Navy training 
school. There are eight (23) combinations of patterns running from 
O00 to 111 (where zeros are incorrect responses and ones are correct). 
A subject having a pattern of 101 has the same number correct as 
Sather subject with the pattern 110, that is, two out of three. 
However, the first indivitital's score might be more predictive of 
success in a particular training school than the second subject's 
binary pattern. 

This study will focus on four different tests or test scores, 


defined as foliows: 
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1. Predictor - the predictor is a long-form test used for 
predicting success in a Navy training school. In this study, the 
predictor is the Electronics Technician Selection Test or the General 
Classification Test. Scores on the predictor are determined by 
counting the items answered correctly. 

2. Criterion - the final school grade in the Basic Electronics 
and Electricity School. 

3. Total Correct - the total number of correct responses out 
of the seven questions selected by SEQUIN analysis for this study. 

4, Pattern Score - a special score assigned to each pattern of 
responses on the same seven items used to compute total correct. 

(A precise definition of pattern scores will be given in Section VE.) 

Thus, the purpose of this study was to: 

1. Gather large pools of data from a Navy training school, 

2. Extract several suitable questions from the General Classi- 
fication Test (GCT) and the Electronic Technician Selection Test 
cE ST); 

_3. Write a computer program that : 
a. constructs all possible patterns of ones and zeros for 
the number of extracted questions 
b. calculates pattern scores for each individual pattern 
C. assigns pattern scores to subjects 
d. correlates the pattern scores of the subjects with 


their final school] grades 
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e. correlates the standard predictor test scores (either 
GCT or ETST) with final school grades 
f. correlates total correct, with final school 
grades , 
g, correlatespattern score with total correct 
h. calculates a multiple correlation coefficient between 
a combination of pattern scores and total correct and 
final school grades 
i. calculates test statistics for the correlations 
j. calculates regression weights for predicting final 
school grades from total-correct scores 
k. creates a frequency distribution showing number of 
subjects with each pattern score 
1. outputs all infor mation in an easy-to-read form for 
use in future studies 
4. Determine those patterns indicative of success for a parti- 
cular training school, 


_o. Test pattern- score predictions by suitable cross- validation. 
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IV. DATA 


AlJ data used in this research were obtained from Mr. Leonard 
Swanson of the Naval Personnel and Training Research Laboratory, 
San Diego, California, and were stored on nine- track magnetic tape 
(Ref. 4). The data consisted of the individual records of approxi-~- 
mately 2400 trainees who started, but not necessarily finished. the 
Navy Basic Electronics and Electricity School in San Diego. Each 
trainee's record consisted of the equivalent of six-computer card 
records listing such information as: 

1. Responses to items on the GCT, ETST, and Arithmetic 
Aptitude Test (ARI) 

Z.) »cores on the GCT, ARI, and ETST 

3. Navy service number 

4, Enlisted rating 

o. Final school grade in Basic Electronics and Electricity 
School 

Tests used as predictors included the GCT and ETST. The GCT 
consists of 60 verbal analogies and 40 sentence-completion items 
with a 35-minute time limit. The ETST consists of three separately 
timed sections: math with 20 items and a 25-minute time limit; 
science with 20 items and a 15-minute time limit; and electricity and 


radio with 30 itéms and a 20-minute time limit (Ref. 5). 
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Two sets of questions were provided by Mr. Swanson along 
with their answer keys. ‘The first set of questions, consisting of 
seven GCT items, and the second set, consisting of seven ETST 
items, were selected using the SEQUIN program, The p-values, 
question types and item validities are shown in Appendixes A and B. 

The criterion consisted of final school grades in the Basic 


Electronics and Electricity School. 


me. DATA PREPARATION 

Two programs were written to extract and put into usable forms 
all pertinent data for the study. (A glossary of terms used in all 
programs is contained in Appendix L.) The first program checked 
for completeness of an individual's record, i.e., the presence of six 
computer-card images, and rejected those subjects whose files were 
deficient. Unfortunately several records contained special characters, 
e.g., dashes, asterisks, etc., instead of integers. Therefore, the 
first data preparation program converted any of these special 
characters to zeros. Thus, a response other than an integer from one 
to five was changed to a zero and counted as an incorrect response. 
If a needed Score such as the GCT, ETST or final school grade was 
blank or contained some non-numerical mark, the record for that 
individual was rejected as being incomplete. (It is possible that some 
of those incomplete records resulted from subjects not finishing the 


school, ji.e., being required to leave the service because of physical, 
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emotional, or academic problems.) The output from this program was 
written on tape or data cell and on paper. 

Appendix C is the flowchart of this first program. Appendix D 
is the program listing. 

Although a subject's record consisted of six computer- card 
records, most information was superfluous. Of the six cards, data 
from three, at most, were considered. Using the answer key supplied 
by Mr. Swanson, the second program graded, on different occasions 
those ETST or GCT questions under consideration. Specifically, it 
assigned a value of one to a correct response and a value of zero to 
an incorrect response. By assigning ones and zeros to the responses, 
the binary pattern was formed. The program also read the criterion 
score and the predictor score. 

The output from this second program consisted of: 

1. binary pattern 

yeeecriterion score 

3. predictor score 

_ 4, an in-house identification number 

ee tie SUD|CEl.s service Umber 

Appendix E is the flowchart for this second program. Appendix F is 


the program listing. 


16 





V. THE MODEL 


The main program is divided into several distinct sections: 
reading of data, determination of a joint frequency distribution, 
computation of pattern scores, assignment of pattern scores to sub- 
jects, computation of correlation coefficients (r's), computation of 
test statistics for r differences, construction of response patterns, 
ordering of response patterns according to the scores computed for 
them, calculation of a multiple correlation coefficient, calculation of 
the correlation coefficient between pattern scores and total correct 
Sererricti on of a frequency distribution showing the number 
of people with each pattern score, and output (printed and 
punched). 


A complete listing of the program is presented in Appendix G. 


je DOUBLE PRECISION REQUIREMENT 
Because of the large sample sizes and the relatively large 
magnitude of several parameters, it was necessary to use double 


precision floating point numbers. 


ae THE DATA CARD 
The data card initializes four variables that are frequently used 
in counting loops (DO loops). The variables, Nl, N2, N3, and N4, 


represent, respectively, the number of people in the sample, number 
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of elements ina pattern, the range of criterion scores, and the 


number of possible binary combinations using N2 items (2N2), 


C, READING THE DATA 

Data is read in only a prescribed format. For this program, 
the individual's data record card is set up as shown in Table I. 

There are two read statements. One read statement Carries 
out the reading of data that is to be used in the computations of the 
program. The other read statement reads a dummy variable, "“IDUM.” 
By placing the read statement involving IDUM before or following the 
main read statement (involving binary pattern, criterion, predictor, 
etc. ), control over alternate selections of data can be attained. For 
example, if odd numbered data were only to be considered, the read 
statement involving IDUM would follow the main read statement thus 
acting as a dummy procedure to contro] data input. Note that all 
input Ae is in FORTRAN " I format." 

The variable "J" is used as the DO LOOP counter involving 
personnel with only one exception. ‘That exception is in the determi- 
nation of the joint frequency distribution. An "I" DO LOOP is used 
for all other counting operations. 

At this point, the total correct out of the extracted questions 
is calculated. The "E" array stores this information. This array is 
used in the calculation of the sum of total correct ron all subjects and 
the sum of the squares of total correct for all subjects. This infor- 


mation is later used in the computation of correlation coefficients 
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and the mean and standard deviation for use in the calculation of a 


test-retest correlation coefficient. 


D. THE JOINT FREQUENCY DISTRIBUTION 

A joint frequency distribution is constructed using decimal 
equivalents of the QN2 binary patterns and the range of criterion 
scores. The rows of the matrix (denoted by matrix variable F) 
represent the decimal equivalents of the binary patterns, and in this 
case there are 128 (2/) binary patterns (the reason for using seven 
questions is explained in the METHODS section). However, the lowest 
binary pattern score (QOQ0000) is also equal to the decimal value zero. 
Therefore, a value of one is added to all decimal equivalents. In 
this way the first row is row one, not zero, and the last row is row 
128. 

The column numbers correspond to successive criterion scores. 
Column one of the matrix corresponds to the subjects’ lowest crite- 
rion score. In this case, the lowest criterion score was 30 and the 
higest was 99. The matrix is represented in Figure l. 

~ The "B"” array is raed to store the decimal equivalent of an 


individual's binary pattern. 


la COMPUTATION OF PATTERN SCORES 
The pattern score for a pattern is the average score of subjects 
who have the pattern and is calculated from the F matrix by tallying 


the number of subjects having the pattern and each criterion score. 
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Figure l 


CONSTRUCTION OF A MATRIX DESCRIBING THE 
JOINT FREQUENCY DISTRIBUTION 


Columns ** 


(30) (99) 
1 69 


Rows* 


*Row numbers are decimal equivalents of binary patterns plus one. 
**Column numbers are criterion scores plus one minus the lowest 
criterion score. Numbers in parentheses are actual criterion 


scores. 
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This number is multiplied by the criterion score and ried. and 
the sum, Sl, is divided by the total number of subjects having the 
Wotteml,. 32. | 

If any of the 128 patterns is not used, because no one has the 
pattern, both Sl and $2 are set equal to zero, and an arbitrary 
score of -1 is assigned to the pattern. 

Immediately following the computation of all pattern scores, 
the scores are outputted on punched cards. The pattern scores 


obtained in this study are presented in Appendix H. 


FB, ASSIGNMENT OF PATTERN SCORE TO SUBJECTS 

A subject's decimal equivalent to his binary pattern is deter - 
mined, and he is assigned the pattern score for that decimal 
equivalent (the row index corresponding to the pattern in the F 


matrix). 


Ce COMPUTATION OF CORRELATIONS 

Correlationcoefficients are then calculated between the 
criterion and the predictor (GCT or ETST) and between the criterion 
and the assigned pattern scores. 

The sums of criterion scores (Cl), pattern scores (XI), and 
predictor scores (Al) are determined along with the corresponding 
sums of squares (C2, X2, A2). The sum of the products of the 
criterion and predictor scores (V), as well as the criterion and pattern 


scores (W), is also determined. The correlation coefficients for 
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pattern vs. criterion (R2) and predictor vs. criterion (Rl) are then 
calculated. The Z test statistic for the difference between these r’s 
is also calculated. | 

Three other correlation coefficients are computed later in the 
program: a.multiple correlation coefficient (see I below), the 
correlation coefficient between total-correct and pattern scores, and 
a test-retest correlation coefficient used as an estimate of the 


reliability of total correct scores on the predictor. 


H. CONSTRUCTION OF RESPONSE PATTERNS 

DICesLIeterdieeh 2s (27) different patterns of responses ranging 
from 0000000 to 1111111, the coroner is assigned the otherwise 
tedious and difficult job of constructing and outputting these patterns. 
A difficulty encountered is that leading zeros of various binary 
patterns, although stored without incident in the machine, are lost 
upon printing. Because of this, all zeros in the binary patterns are 
converted to twos. This fact is noted on the printed output (Appendix 


H). 


I. MULTIPLE CORRELATION COEFFICIENT 

The multiple ocrrelation coefficient indicates the eta of 
relationship between one variable and a linear combination of two or 
more others that produces the strongest relationship. Since different 
predictor variables are sometimes intercorrelated and so duplicate 


one another, the multiple correlation coefficient depends on the 
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intercorrelation of different predictor variables as well as on the 
correlation of each with the criterion variable (Ref. 6). 

Specifically, the multiple correlation between criterion scores 
and a combination of pattern scores and total-correct scores is 
computed. 

Since the coefficient of multiple correlation considers the 
inter- relationship between the predictor variables, it should have, 
theoretically, a greater value than the correlation between either 
predictor and final school grades alone. 

The significance of the multiple r is next computed using an F 
statistic where F is the ratio of the variance of the residuals on the 
criterion before considering the multiple correlation coefficient and 


the variance of the residuals after consideration. 


- COMPUTATION OF REGRESSION WEIGHTS 

Since there is a possibility that some binary patterns will not 
be used (i.e. , there may be some binary patterns no one has because 
the sample size is small in relation to the number of binary combi- 
nations), it is conceivable that an individual in the cross- validation 
eroup en have a pattern that no one in the validation group 
has. Correspondingly, regression weights are computed from the 
relation between total-correct and criterion scores in the validation 
group that are to be used as input in the cross-validation study to 


determine scores for individuals having pattern scores equal to -1. 
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K. OUTPUTS I AND II 

The pattern responses are then sorted according to pattern 
score from the lowest (-1) to the highest (73. 66) and, in conjunction 
with the pattern score and total correct of that binary pattern, are 
printed out in tabular form. The table and results thus obtained are 
shown in Appendix H as Output I. 

Next, tables are prepared listing the subject's in-house identi-. 
fication, his predictor score, his final school grade (criterion score), 
the pattern score associated with his binary pattern, and, finally, 
the total correct scored out of the seven questions. A sample showing 


the first 50 subjects is presented in Appendix I as Output II. 


bse ADDITIONAL OUTPUT 
All correlations and test statistics computed during the 
execution of the program are also printed. These results are pre- 


sented and discussed in the RESULTS section. 
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VI. CROSS- VALIDATION 


Cross- validation is a method used to estimate the magnitude 
of sampling variation. In cross-validation, results are obtained 
from a second sample of people for comparison with the results of 
an initial sample. If the results obtained from the eecone sample 
confirm the results of the first the results are said to hold up under 
cross- validation. 

In addition to the validation or main program, described in the 
preceding section, this study makes use of a cross-validation program, 
which is essentially a portion of the main program. It differs in that 
pattern score and regression weights derived from the previous 
program are readinwithnew subjects’ personal data and that patterns 
are not constructed, pattern scores are not calculated, and there is 
no need for a joint frequency distribution. The program listing for 
the cross-validation is presented in Appendix J. 

As can be seen from Appendix H, there are fourteen binary 


patterns that were not used by the validation group in the ETST study. 


' Therefore, the cross-validation program has to determine if a subject 


has a pattern that was not used in the validation program and, if he 


has, it must assign him a score using the regression weights deter- 


mined from the validation group and his total-correct score. 
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ioe ASSIGNMENT OF PATTERN SCORES 

Various other methods of assigning pattern scores to patterns 
that no individual in the validation group has were attempted. These 
methods included: using the average pattern score derived from the 
main program, weighting more heavily those patterns appearing 
more frequently than those appearing less frequently, ignoring a 
subject in the cross-validation who had a pattern no one had in the 
validation group (with adjustment of corresponding variables, e.g., 
sample size), and finally using the regression weights. 

With only one exception, that of using the regression weights, 
all methods of attack failed. All pattern-score validities were 
significantly lowered in all the other cases. (The reason for the 
abrupt drop in pattern- score correlation coefficients in the cross- 
validation is discussed in the RESULTS section: ) 

Using the regression weights, however, pattern score validi- 
ties Maintained a maximum, Scores were obtained by adding the 
product of total correct and the slope regression weight to the 
Be OTe C COG nia ii, 

Inputs for the cross-validation consisted of the same information 
as noted in the main program plus the regression weights and the 
pattern scores from the main program. 

The tabular results for the first fifty subjects (even numbers 


only) is presented in Appendix K as Output III. 
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VU. METHOD 


The validation and cross-validation programs were first used 
on GCT data. Not only was the GCT data analyzed, but it also 
served, at the beginning of the research effort, as a test platform for 
debugging the validation and cross-validation computer programs. 


The study concentrated on the ETST data, however. 


A. PRELIMINARY STUDIES 

GCT data were used in preliminary studies. Use of GCT data 
as a predictor, as originally planned, was unsatisfactory because 
the GCT was not designed as a predictor of success in a training 
school and, of the seven questions considered in the study, approxi- 
mately one-third of the sample subjects had all correct, which is 
hardly an indication of predictive validity. 

The first step in the study was the determination of the sample 
size to be used in the validation and cross-valididation programs. 
Since the total number of possible combinations of ones and zeros 
was 128 (27), it was decided that an appropriate sample size in the 
main program would be 2,000. This would result in the theoretical 


utilization of 15-16 subjects per binary pattern: 


2,000 subjects _ 
128 patterns ~ fos? 
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The remaining subjects in the sample (379) would then be used in 
cross-validation studies. 

Table I] summarizes the results of this first effort. As can be 
seen from these results, the greatest validity for the validation 
(main) group was obtained from the predictor vs. criterion scores 
(r =Q.51). However, the relationship between the pattern and 
criterion scores was only 0.44. Although lower than the predictor 
validity coefficient, it was still better than the r for total correct 
(0.37). The high absolute values of the test statistics indicate that 
all the differences were significant. 

The cross validation tells essentially the same story. The 
validities for the pattern and total correct were very 
nearly the same as in the validation program. However, the validity 
for the patterns fell short of its counterpart in the validation program 
(Tyva] = .34 vs. Fyq] = .44). This phenomenon resulted from the 
weighting of item responses which maximized the correlation with 
the criterion, i.e., minimized the error of prediction, thereby 
capitalizing on chance in the validation group. ‘The fact that chance 
played an important role in the validation program was further 
illustrated by one subject who had a binary pattern with four ones, 
i.e., four out of seven correct, but who also had the highest of all 
pattern scores. 

Another explanation for the substantial reduction in pattern 


vs. Criterion validities is that the mean pattern score was used for 
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CORRELATION COEFFICIENTS AND 
TEST STATISTICS DE RIVED FROM 
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Main 


Cross 
Validation 










Te The first Z is for the difference between the pattern- 


criterion and the predictor-criterion correlations. The second Z 


is for the difference between pattern-criterion and total correct- 


criterion correlations. 


2. The sample size in the main study was 2,000 subjects 


while the sample size in the cross-validation study was 379 subjects. 
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individuals in the cross-validation group who had patterns no one had 


in the validation group. 


B. ASSIGNMENT OF PATTERN SCORES 

Because of the discrepancies in pattern-score validities for 
both the validation and cross-validation programs, the problem of | 
assigning a valid pattern score to an individual who, in the cross- 
validation process, had a pattern no one had in the main validation 
arose. ‘Therefore, the several] approaches mentioned earlier were 
formulated arid attempted. 

1. The First Solution 

The first of these proposed solutions involved the use of 

weights proportional to the number of subjects having a pattern. 
The weights were to be calculated, along with the pattern scores, 
in the main program and outputted on punched cards. ‘The theory 
behind this solution was that if a binary pattern appeared very 
frequently it should have been counted more heavily in the cross- 
validation than those patterns appearing less frequently. Once again, 
considering the subject who had the highest pattern score with only 
four oe it would appear logical that that person was not typical 
and should not have been counted equally as others. That is, would 
it have been valid to give his score the same weight as a score 
that was 25 per cent more prevalent? If both scores receive equal 
weight, distortion of the validities must certainly occur. Unfortunately, 


a suitable method of computing and applying such weights was not found. 
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2. The Second Solution 
The second solution was to reduce the number of questions 
used in the study from seven to six. With only six questions the 
number of binary combinations would have been significantly reduced 
(from 128 to 64) resulting in the utilization of more binary patterns. 
It was hoped, in fact, that all binary patterns would have been used. 
Thus, when going into the cross-validation phase all patterns would 
have had pattern scores and the need for generating pattern- score 
substitutes in the cross-validation could have been eliminated. 
However, even with consideration of only six questions (64 combina- 
tions of ones and zeros), eight binary patterns were not used. 
Furthermore, the magnitudes of the correlation coefficients decreased 
markedly. Therefore, this approach was eliminated. 
3. The Third Solution 
The third solution called for the elimination of those 
subjects in the cross-validation who had a binary pattern no one 
had in the validation study. The theory behind this solution was, in 
essence, to eliminate the problem by pretending ft wasn't there. 
This solution was not suitable for apparent reasons. For.a test to 
be valid in a real environment, vis a vis a laboratory environment, 
it must consider all contingencies. 
4, The Final Solution 
It was finally decided to calculate regression weights and 


use these in assigning pattern scores to subjects in the cross-valida- 


tion who had patterns no one had in the main program. 
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C. READ IN OF ALTERNATE DATA 

The possibility of sample bias was also considered, e.g., 
predictor or criterion scores of the entire sample could have been 
placed inorder of increasing or decreasing magnitude. Therefore, 
it was decided to split the sample in half; the first half to be used 
in the validation program and the second half in the cross-validation 
program. ‘The main or validation program was then designed to 
read the records of every alternate subject, e.g., every odd- 
numbered subject, and make appropriate calculations from those 
data. § The cross-validation also read every alternate 
but complementary record. Thus, for example, if the main program 
read every odd record, the cross-validation program Beeepotinaly 
read every even record. Unfortunately, however, splitting the 
sample this way See rited in a drop in the number of 


subjects per binary pattern from fifteen to approximately nine. 


D. rie 2 sas | Dian 

PAE UC I solving the problem of assigning pattern scores to 
subjects in the cross-validation study, the research focused on 
utilization of the ETST as the predictor. 

Data preparation followed the same procedures as those 
noted in the data-preparation section of this thesis. 

In addition to the tables and correlation coefficients computed 
in the validation and cross-validation processes, the programs also 


outputted the sum of total correct and the sum of the squares of 
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total correct for all subjects. This was used in the computation of 


the mean and variance for total correct (total ones). The reason 
for these calculations was to determine the test-retest correlation 
coefficient. 
1. Test- Retest Reliability Coefficient 

The test-retest reliability coefficient, as described by 
Weitzman (Ref. 7), is an estimate of the correlation between identical 
versions of a test taken by the same persons in independent trials. 
For a test with n-items and a-alternatives, this estimate is: 


ia ] - n-M 
aS2 (1) 


where M and S are the mean and standard deviation, respectively. 

This estimate of the test-retest reliability coefficient 
can be used in the determination of the correction for attenuation. 

Pa COLPection for Attenuation 

Because correlation results are obtained from fallible 
measurements, errors tend to reduce or attenuate the correlation 
between traits. Using the formula for correction for attenuation, it 
is possible to estimate what the correlation would be if De hier 
errorless measurements were available (Ref. 8). Correlation 
coefficients that are corrected for attenuation cannot be used in pre- 
diction equations but can be used when analyzing relationships to 
make allowances for random errors of measurement. 


Using the test-retest correlation coefficient computed for 


the predictor, it is possible to calculate the validity of the predictor 
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corrected for attenuation. The value obtained from the following 
formula is the theoretical correlation coefficient if the predictor 


were error-free: 


AY pr (2) 
The correlation coefficient rpc is the measured validity between 
the predictor and criterion, and pp is the test-retest reliability 
described in the previous section. A comparison between the 
validity coefficient (Tpc) and the validity coefficient corrected for 
attenuation (rq, c) was used as an indication of how close this study 
came to the theoretical limit of validity for the predictor. Specifi- 


cally, r,_, (total correct vs. final schoo] grade) was compared to 


Be 
the corresponding correlation coefficient corrected for attenuation. 
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VIII. RESULTS 


ee DETERMINATION OF LINEARITY 

A product-moment correlation coefficient is good only. if a 
linear relationship exists between the variables that are being corre- 
lated. Figures 2 and 3 are scatter diagrams which were used to 
determine if a linear relationship existed between total correct out 
of seven and final school grade and the full ETST score and the 
final school grade. Note that almost all the points can be enclosed 
in an oval which goes from the lower left to the upper right, there- 


fore indicating linearity (Ref. 9). 


B. CORRELATION COEFFICIENTS AND TEST STATISTICS 
Table III lists the values for all test statistics and correlation 
coefficients. 


As can be seen from that table, the value of Tyattern score 


decreases from 0. 76 in the validation program to OQ. 72 in the 
‘cross- validation program, the reduction due to maximization of 
chance in the main program. ‘The computation of the multiple 
correlation coefficient was desired to see if pattern scores add to 
the predictive ability of total-correct scores. .The 

multiple correlation coefficient did not increase the value of 


Ytotal correct thus indicating no additional predictive ability. The 


36 


FINAL SCHOOL GRADE (CRITERION) 


00 


el Oe eee 


55 60 65 
PREDKTOR SCORE (ETST EXAM) 
SCAT LER ELOl 


37 





0 





FINAL SCHOOL GRADE (CRITERION) 


prow e = S 


OO OO 6OQ 65 


PATTERN SCORE (ASA PREDICTOR) 
SCATTER WEOT 


38 





TQ 





——— fae een eee kl 


TABLE III 


7 CORKELEA TION COBFFPFICIE Nis waND 
TEST STATISTICS DERIVED FROM 
THE ELECTRONICS TECHNICIAN 
See CoION SES 






Cross- 
Main Validatnon 


r(pattern score) } 0.7 Ovei2 
r(predictor) 0. 61 0. 60 

Ve 713 4.93 
r(total correct) | 0.72 Oe 
Ze 2. 26 4 
r(pattern score / 

total correct) | 0.95 

5 
3 


OR 
r(multiple) 0.76 — 0.7 
R 176. 36 one o 

Che 












Correction for 
Attenuation O 


o 
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1. The first Z is for the difference between the pattern- 


criterion and the predictor-criterion correlations. The second Z 


is for the difference between pattern-criterion and total correct- 


criterion correlations. 


2. The sample sizes in both the main and cross-validation 


studies was 1,182 subjects. 
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large value of PF indicates that the total-correct scores contributed 
significantly to the predictive ability of the pattern scores, 
however. 

The high value of the correlation coefficient between pattern 
scores and total-correct scores indicated that the seven items used 
in the study constituted a very valid test and that the total correct 
could be used as a predictor that is as good as the pattern scores 
for these items. 

The correction for attenuation revealed that the highest validity 
theoretically obtainable by improving the reliability of the seven- 
item predictor was 0.85, ‘The value actually obtained, 0.73, was 
equal to the test-retest reliability of the test. Since it is not reason- 
able to expect that a test will correlate more highly with another 
test thah it does with itself, it is no wonder that the pattern scores 


did not correlate better with the criterion than the total-correct did. 
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IX. CONCLUSION 


The two FORTRAN computer programs developed in this 
study successfully determined and correlated pattern scores with 
the criterion. However, the questions extracted from the ETST 
were so highly valid that they could have been used alone, i.e., ” 
without pattern scoring, as predictors of success in the Basic 
Electronics and Electricity School. 

It would be interesting to continue this study using biographical 
information, not ordinarily quantifiable, instead of extracts from 
current examinations. Biographic&él questions carefully constructed 
and easily verifiable could be used in predicting behavior, and pattern 
scoring is a method that can be used to quantify responses to these 
questions. Responses quantified by pattern scoring, in fact, will 


show the highest possible correlations with predicted behavior. 
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APPENDIX A 


ieee vetspccimene lh [lenis sclectedsay sROULN 


|) ee) ee () ee (4) (5) 


Form 7 Item Recruit Median Median 
Item Type p Value School School 
Number p Value Validity 
13 A swicd: oo oD 
19 SC . 60 78 220 
3k A ie AOO 20 
oe) A 41 49 . 24 
62 SC . 60 . 69 526 
67 HC . 80 .8/ 720 
94 SC oo io 30 

NOTE: 1. Values in Columns (4) and (5) are based on item data 


only for schools in which that item was selected in Program SEQUIN. 


2. A= analogies; SC = sentence completion item. 
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Question 
Number 
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APPENDIX B 


Wie Seveibeote Is iiteme selected by SEQUIN 


{tem 
Type* 


M 


M 


Recruit 
P- Value 


.o/ 
. 38 
08 
5 oy 
mo 
eo 


- Re 


Median School 
P- Value 


7 
06 
oo 
a 
4 8 
5 ok, 


ol 


M = Math; S = Science; E = Electricity or Radio 
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APPENDIX E 


SECOND DATA PREPARATION PROGRAM 
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Listing of Validation Program 
(Continued) 
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Listing of Validation Program 
(Continued) 


PATTERNS 


3 
rom 


A 

> 

N 

) 
fripoGecsty hh. GO 10-80 


e 

E 

u 

xk 

5 
oi) *RISTAR 
E.N4) GO TO 60 


I=1,N 


mee TAS AO 
SC SOMO ~aNiW 
FOW+ONM{ * 
asus tel ONY NS 
wt OO ee SS ee 
Hope eT UD set 
mt ra LL QIN AO TE LL 
NY NIRA AY INA IN eS 


fOTRWO 
mt tl ee UY 
NOre Ow enmem Os 

Woe thay ZZ Off. ora ILO 
em me HOW + + mee 
SSeS Nwe SRS 

net eet es er |||] Lf] Cer eer eer LL) 
eS Leas SOO OOS 


eEQ.1) GO TO 70 


{ 

{ 

{ 
NUE 


$ ORR HH ead mY 
ZO OF NOD RH He eHs 

Te a ee | eS ee | a 
Za Z2O> BSS SENOLMNOL~ OR 


N 
Q 
Sone Or Toi E Pe UCST SPATISTIC FOR R OLEFERENCE 
LU sey 
CENS T.UCTION OReRESPONSE PATTERNS 
N 
) 
) 
2 
ORDERING OF RESPONSE 


fe Nels 7 


Oo = — ore) Oo 
uy uy Or oO 
OONU OOOO QHQUGes OOOO Qouoe 


20 





Listing of Validation Program 
(Continued) 
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eee 
GEOSSAKY OF 


COMPUTE R VARIABLES 
USED IN THESE PROGRAMS 


A(J) - jth subject's GCT score used as a predictor 


Al - sum of predictors 

A2 - sum of squares of predictors 

B(J) - jth subject's decimal value of his binary score 

C(J) - jth subject's final school grade used as the criterion 
Cr = sum of the criterion scones 

vee Sum oO; SQuares Of Criterion score 


DJ) - jth subject's identification number 
E(J) - jth subject's total correct 
F(,) - joint frequency distribution 


GC ) binary pattern (2 replaced O in output) 


H( ) total correct in a binary pattern 
IAI - sum of total correct (total ones) 
WAZ - sum of square of total correct (total ones) 
IHI 

- used in calculating range of criterion scores 
ILO 
IV. - sum of C(J)*E()) 


IW( )- a column on a subject's record card 
KDNUM - card number 


M  -. column in 'F matrix’ 
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Glossary of Computer Variables Used in These Programs 


(Continued) 
N  - row in ‘F matrix’ 
Nl - sample size 
N2 - number of elements in the binary pattern 
N3 - range of criterion scores 
N4 - 2**N2; number of combinations of patterns of 1/0 with N2 
questions 
P(,) - jth subject's pattern of ones/zeros 
Rl - correlation coefficient between criterion/predictor 


2nd time corelation coefficient between criterion/total correct 
R2 - correlation coefficient between criterion/pattern 
R3-5- correlation coefficients used in determining Rl and R2 
R7 - correlation coefficients between pattern scores and total ones 
RMUL - multiple correlation coefficient 
s(I) - a pattern score associated with a particular pattern 


Sl  - weighted sum of people with that pattern (weights being the 
criterion scores) 


D2 eeeeenurmber of people with that pattern 
V ee tot product of C(J)*A(J) 
W  - sum of product of C(J)*X(J) 


W(1-7) - an array of questions being used in this study 
X(J) - jth subject's pattern score 
Xl - sum of pattern scores 


X2 - sum of square of pattern score 
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Glossary of Computer Variables Used in These Programs 
(Continued) 


fa - test statistic 


F = test statistic F distribution 
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